Merging and stability for time inhomogeneous finite Markov chains

As is apparent from most text books, the definition of a Markov process includes, in the most natural way, processes that are time inhomogeneous. Nevertheless, most modern references quickly restrict themselves to the time homogeneous case by assuming the existence of a time homogeneous transition function, a case for which there is a vast literature. The goal of this paper is to point out some interesting problems concerning the quantitative study of time inhomogeneous Markov processes and, in particular, time inhomogeneous Markov chains on finite state spaces. Indeed, almost nothing is known about the quantitative behavior of time inhomogeneous chains. Even the simplest examples resist analysis. We describe some precise questions and examples, and a few results. They indicate the extent of our lack of understanding, illustrate the difficulties and, perhaps, point to some hope for progress. We think the problems discussed below have an intrinsic mathematical interest (indeed, some of them appear quite hard to solve) and are very natural. Nevertheless, it is reasonable to ask whether or not time inhomogeneous chains are relevant in some applications. Most of the recent interest in Markov chains is related to Monte Carlo Markov Chain algorithms. In this context, one seeks a Markov chain with a given stationary distribution. Hence, time homogeneity is rather natural. See, e.g., [26]. Still, one of the popular algorithms of this sort, the Gibbs sampler, can be viewed as a time inhomogeneous chain (one that, despite huge amount of attention, is still resisting analysis). Time inhomogeneity also appears in the so-called simulated annealing algorithms. See [12] for a discussion that is close in spirit to the present work and for older references. However, certain special features of each of these two algorithms distinguish them from the more basic time inhomogeneous problems we want to discuss here. Namely, in the Gibbs sampler, each individual step is not ergodic (it involves only one coordinate) whereas, in the simulated annealing context, the time inhomogeneity vanishes asymptotically. Other interesting stochastic algorithms that present time inhomogeneity are discussed in [10]. In many applications of finite Markov chains, the kernel describes transitions between different classes in a population of interest. Assuming that these transition probabilities can be observed empirically, one application is to compute the stationary